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LONG-TERM  GOALS: 

Develop  an  advanced  global  atmospheric  forecast  system  designed  to  exploit  massively  parallel 
processor  (MPP),  distributed  memory  computer  architectures.  Future  increases  in  computer  power 
from  MPP’s  will  allow  substantial  increases  in  model  resolution,  more  reabstic  physical  processes,  and 
more  sophisticated  data  assimilation  methods,  all  of  which  will  improve  operational  numerical  weather 
predictions  and  provide  better  simulations  of  the  Earth’s  climate. 

OBJECTIVES: 

The  current  Navy  operational  global  atmospheric  prediction  system  (NOGAPS  3.4)  is  a  highly 
optimized  Fortran  code  designed  to  run  on  parallel  vector,  shared  memory  machines  (CRAY’s).  The 
prototype  new  system  is  NOGAPS  4.0.  The  immediate  objective  of  the  project  is  to  redesign  the 
model’s  numerical  algorithms  and  data  structures  to  allow  efficient  execution  on  MPP  architectures 
and  clusters  of  shared  memory  processors.  Message  passing  (MPI)  is  the  paradigm  chosen  for 
communication  between  distributed  memory  processors.  This  work  is  support  by  ONR  Marine 
Meteorology,  PE  0602435N  (035-71). 


APPROACH: 

Use  control  integration’s  of  the  current  operational  NOGAPS  to  ensure  reproducibility  of  results  with 
the  newly  designed  Fortran  90  code.  Design  efficient  spectral  transform  algorithms  for  both  shared 
memory  and  distributed  memory  architectures.  For  distributed  memory  architectures  use  message 
passing  library  modules  in  communication  intensive  spectral  transforms  and  horizontal  interpolation 
routines. 

WORK  COMPLETED: 

The  Fortran  90  version  of  NOGAPS  4.0  has  been  multi-tasked  to  run  on  both  the  CRAY  90  and  the  in- 
house  6-processor  DEC  Alpha  SMP.  The  code  complies  with  all  Fleet  Numerical  Meteorology  and 
Oceanography  Center  (FNMOC)  and  DOD  documentation  requirements.  Original  plans  were  to  use 
High  Performance  Fortran  (HPF)  as  the  mechanism  for  porting  NOGAPS  4.0  to  distributed  memory 
architectures.  However,  immature  HPF  compilers,  disappointing  performance,  and  general  lack  of 
HPF  support  in  the  meteorological  modeling  community  has  lead  us  to  switch  to  explicit  message 
passing  instead.  MPI  is  now  the  de-facto  standard  for  distributed  memory  architectures  and  is  well 
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supported  by  virtually  every  computer  vendor.  We  now  have  an  MPI  version  of  NOGAPS  4.0  running 
in  test  mode  on  the  DEC  Alpha.  The  spherical  harmonic  transforms  have  been  ported  to  several 
distributed  memory  architectures  to  ensure  scalability,  portability,  and  robustness  of  the  MPI  code. 

RESULTS: 

The  MPI  version  of  NOGAPS  4.0  is  now  running  on  the  DEC  Alpha  SMP.  The  operational  model, 
NOGAPS  3.4,  is  being  used  as  a  control  to  ensure  reproducibility  of  results.  In  spite  of  radically 
different  algorithm  design  for  the  spherical  harmonic  transforms,  8-10  decimal  place  reproducibility  is 
being  achieved  for  12  hour  forecasts.  Virtually  all  inter-processor  communication  is  concentrated  in 
the  spherical  harmonic  transforms.  The  figure  below  shows  T3E  scalability  results  for  a  wide  range  of 
processor  numbers  and  model  resolutions.  The  top  line  corresponds  to  a  T213,  32  level  model,  the 
bottom  line  a  T21  model.  A  straight  line  indicates  perfect  scalability  from  4  to  128  processors.  The 
higher  resolution  cases  achieve  this  goal.  The  lower  resolution  results  indicate  there  is  insufficient 
‘granularity’  to  give  good  scalability,  especially  for  the  128  processor  cases.  However,  because  these 
results  are  for  the  transforms  only,  we  can  expect  much  better  scalability  for  the  complete  model  when 
the  embarrassingly  parallel  diabatic  processes  are  added. 


T3E  SPEC TRAL  TRANSFORM 


IMPACT: 
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NOGAPS  is  run  operationally  by  FNMOC  and  is  the  heart  of  the  Navy’s  operational  weather 
prediction  support  to  nearly  all  DOD  users  worldwide.  It  is  also  run  by  many  NRL  and  other  Navy 
researchers  to  study  atmospheric  dynamics,  atmosphere/ocean  interaction,  and  climate  variability. 
NOGAPS  4.0  is  the  next  generation  of  this  system  being  targeted  for  the  next  generation  of  computer 
architectures.  These  architectures  are  expected  to  be  distributed  memory,  commodity  based  systems 
with  enormous  theoretical  computational  power.  However,  exploiting  this  capability  will  require 
drastically  redesigning  many  important  model  algorithms. 

TRANSITIONS: 

Improved  algorithms  for  model  processes  will  be  transitioned  to  6.4  (PE  0603207N)  as  they  are  ready, 
and  will  ultimately  be  transitioned  to  FNMOC  with  future  NOGAPS  upgrades.  Development  of  the 
MPI  NOGAPS  code  has  necessitated  close  examination  of  the  algorithms  used  in  the  operational 
model,  and  in  some  cases  uncovered  design  weaknesses  and  bugs  that  are  being  promptly  corrected  in 
the  operational  NOGAPS. 

RELATED  PROJECTS: 

(1)  NOGAPS  4.0  Evaluation  (X05 13-01):  Advanced  development  and  transition  of  the  NOGAPS  4.0 
forecast  model  to  operational  status  at  FNMOC.  (2)  The  DOD  CHSSI  Scaled  Software  algorithm 
development  for  meteorological  models  (HPCM-96-032):  Development  of  numerical  algorithms 
appropriate  for  massively  parallel  computer  architectures.  These  algorithms  will  be  critical  for  inter¬ 
processor  communication  dependent  and  computationally  intensive  model  processes.  An  important 
component  of  this  effort  is  to  have  the  MPI  version  of  NOGAPS  available  by  3Q  FY98  for  use  as  a 
benchmark  code  for  use  in  the  FNMOC/CNOC  computer  procurement. 
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